Many security activities and military operations take place in complex environmental conditions. The operations and activities include, for example, surveillance, robotic navigation in uneven terrain, rescue missions in uneven terrain or debris, identification of live or dead bodies, and intruder detection in critical areas. The environmental conditions include a variety of weather conditions, such as rain and fog, smoke and different light conditions based on the time of day. To efficiently and effectively perform these activities in the variety of environmental conditions, it is necessary to have a sophisticated level of situational awareness. To achieve the desired level of awareness of the surroundings while providing an acceptable level of visualization, a system is needed that is multispectral and omnidirectional.
Multispectral imaging involves a collection of two or more monochrome images of the same scene, each taken with a different sensor, wherein each sensor captures a different wavelength band of the electromagnetic spectrum. Each sensor creates an image in each separate wavelength band or spectra. The images in each spectra are then combined to construct a multispectral image. Since the multispectral image includes images created from spectra other than visible light, the multispectral image provides a clearer and higher resolution image to an operator.
The concept of multispectral image analysis has been used since the 1960s, specifically in satellite imaging applications. For example, the Landsat series of Earth satellites capture four images at different spectral bands. More recent sensors, know as hyperspectral sensors, provide hundreds of spectral samples in the mid to far infrared wavelength range. For example, two well-known hyperspectral sensors, the Hyperspectral Digital Imagery Collection Experiment (HYDICE) sensor (Basedow et al. (1995) Proceedings of SPIE 2480:258–267), and the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) (e.g., see Vane et al. (1993) Remote Sensing of the Environment 44:127–143; Wheeler et al. (1999) Photonics Spectra 33(4): 124), each provide over two hundred spectral samples at each image location.
The multispectral sensing of high altitude images allows the analysis of aerial images with a high degree of precision. For example, using multispectral imaging analysis, aerial images include such detailed visualization as material classification, rock/soil analysis, plant monitoring, and airborne mine detection. However, although additional spectral information has been used to improve image analysis in aerial imaging, multispectral imaging, using more than two spectra, has not been used in analyzing non-aerial images.
Along with the resolution required for effective visualization and effective awareness of the surroundings, a system is required for viewing a wide field of view. Traditionally wide-angle lenses are used to capture a large field of view, (see, e.g., relevant pages in Panoramic Vision (Benosman et al., Ed.), Springer Verlag, 1st edition (2001); Proceedings of IEEE Workshop on Omnidirectional Vision, Daniilidis (program chair); sponsored by IEEE Computer Society (Hilton Head Island, S.C., Jun. 12, 2000)). However, such lenses, also called fish-eye lenses, are complex combinations of individual lens elements and these lenses introduce radial distortions that are hard to model. Consequently, rotating cameras are currently used to provide visualization for a wide field of view. A special transformation, independent of the scene, is used for mapping images from a purely rotating camera. This transformation enables the mosaicking of the images into a panoramic image. Several mosaicking systems exist based on purely rotating cameras (see e.g., Szeliski (1996) Computer Graphics and Applications, 16(3):23–30).
However, even though the mosaicking systems create high quality panoramic images, the systems are more suitable for photographic purposes. These existing mosaicking systems using rotating cameras to provide panoramic images have three main weaknesses: (a) simultaneous, multi-directional image acquisition is impossible; (b) they can only be successful with stationary scenes; and (c) they need additional power for the motors moving the cameras. Consequently, these systems are not suitable for the dynamic applications, such as navigation, and the continuous alertness required for surveillance.
Omnidirectional systems comprise devices for visualization of hemispherical fields of view. Omnidirectional imaging is based on combinations of mirrors and lenses, such as those used in telescope construction. Initially, the omnidirectional systems used a panoramic field of view camera (U.S. Pat. No. 3,505,465 (Rees)). Subsequently a pyramidal conical mirror-lens system was introduced (Nalwa, (1996) Technical report, Bell Labs, Holmdel, N.J.). Finally, the use of a Head Mounted Display (HMD) was developed (Boult, IEEE Conf. Computer Vision and Pattern Recognition, pp. 966–967, Santa Barbara, Calif., Jun. 23–25, 1998). When using an HMD, a tracker provides the orientation of the observer's head and maps the section of the panoramic image to the position of the observer's head, so that the image matches the position of the observer's head. In this way, the operator using a HMD continues to accurately view the scene as the operator's head is rotated. However, these trackers can be used by an operator only in a closely controlled environment.
Accordingly, the current systems have failed to provide adequate sensing systems for operations that may be used at any time of day in smoky, rainy, and foggy conditions. Thus, a need has remained for a means for effective and efficient visualization and surround awareness necessary for security and robotics applications through combining multispectral imaging for improved visualization and an omnidirectional system for viewing a wider field of view.